Three-dimensional face recognition

ABSTRACT

Apparatus for obtaining 3-Dimensional data of a geometric body for matching, and particularly for use in facial matching, comprises a three dimensional scanner for obtaining three-dimensional topographical data of the body, a triangulator for receiving or forming said data into a triangulated manifold, a geodesic converter, for converting the triangulated manifold into a series of geodesic distances between pairs of points of the manifold, and a multi-dimensional scaler, for forming a low dimensional Euclidean representation of the series of geodesic distances, to give a bending invariant representation of the geometric body. In one variant, matching is carried out by taking the principle eigenvalues from the representation and plotting as co-ordinates in a feature space. Tilted or different expression versions of the same face tend to form clusters in the feature space allowing for matching. The apparatus preferably uses the fast marching method for the triangulated domain to obtain the geodesic distances.

RELATIONSHIP TO EXISTING APPLICATIONS

The present application claims priority from U.S. Provisional PatentApplication No. 60/416,243 filed Oct. 7, 2002, the contents of which arehereby incorporated herein by reference.

FIELD AND BACKGROUND OF THE INVENTION

The present invention relates to a method and apparatus forthree-dimensional face recognition and, more particularly, but notexclusively to such a method and apparatus that both obtains a 3Drepresentation of a face and uses that representation for matchingpurposes.

Face recognition has recently become an important task of computervision, and is needed in a wide range of biometric and securityapplications. However, most existing face recognition systems arereported to be sensitive to image acquisition conditions such as headposition, illumination, etc. and can therefore be inaccurate and easilyfooled. Reference is made to American Civil Liberties Union (ACLU) ofFlorida, Press Release, 14 May 2002, Available:http://www.aclufl.org/pbfaceitresults051402.html.

In general, modern face recognition approaches can be divided into twowide categories: 2D approaches, using only image information (which canbe either grayscale or color), and 3D approaches, which incorporatethree-dimensional information as well.

While simpler in data acquisition (which permits real-time surveillanceapplications, such as face recognition from a video-taped crowd in pubicplaces), the 2D approach suffers from sensitivity to illuminationconditions and head rotation. Since the image represents the lightreflected from the facial surface at a single observation angle,different illumination conditions can result in different images, whichare likely to be recognized as different subjects (see FIG. 3). One ofthe classical 2D face recognition algorithms is the Turk and Pentlandeigenfaces algorithm. For a full discussion see M. Turk and A. Pentland,Face recognition using eigenfaces, CVPR, May 1991, pp. 586-591, and M.Turk and A. Pentland, “Face recognition system, U.S. Pat. No. 5,164,992,17 Nov. 1990. The Eigenfaces algorithm works as follows: Given a set offaces arising from some statistical distribution, the principalcomponents of this distribution form a set of features that characterizethe variation between faces. “Eigenfaces” are the eigenvectors of theset of “all” faces. The eigenface approach treats face recognition as a2D classification problem, without taking into consideration thesignificant difference in the images resulting from illuminationconditions and head rotation. For this reason, eigenfaces usuallyproduce mediocre results when faced with real life rather thanlaboratory conditions.

The 3D approach provides face geometry information, and face geometryinformation is independent of viewpoint and lighting conditions. Thussuch information is complementary to the 2D image. 3D information,however, not only carries the actual facial geometry, but includes depthinformation which allows easy segmentation of the face from thebackground.

Gordon showed that combining frontal and profile views can improve therecognition accuracy and reference is made to G. Gordon, “Facerecognition from frontal and profile views”, Proc. of the InternationalWorkshop on Face and Gesture Recognition, Zurich, Switzerland, pp 47-52,June 1995.

Beumier and Acheroy show the adequacy of using geometric information inthe rigid face profile for subject identification in a system usingstructured light for 3D acquisition, and reference is made to C. Beumierand M. P. Acheroy, Automatic Face Identification, Applications ofDigital Image Processing XVIII, SPIE, vol. 2564, pp 311-323, July 1995.

The above described approach may be generalized to the whole surface,and reference is made to C. Beumier and M. P. Acheroy, Automatic FaceAuthentication from 3D Surface. British Machine Vision Conference BMVC98, University of Southampton UK, 14-17 Sep. 1998, pp 449-458”, 1998,who describe such a generalization using global surface matching.However, surface matching is sensitive to facial expressions and cannotbe considered a comprehensive solution.

There is thus a widely recognized need for, and it would be highlyadvantageous to have, a facial recognition system that usesthree-dimensional information but is devoid of the above limitationssuch as being sensitive to facial expressions, lighting of the subject,or to angle.

SUMMARY OF THE INVENTION

The present invention obtains a canonical form representation of 3-Dfacial data and uses that canonical representation to compare with otherfaces. According to one aspect of the present invention there isprovided apparatus for processing 3-dimensional data of a geometric bodyfor matching, said apparatus comprising:

a geodesic converter, for receiving 3-dimensional topographical data ofsaid geometric body as a triangulated manifold, and for converting saidtriangulated manifold into a series of geodesic distances between pairsof points of said manifold, and

a multi-dimensional scaler, connected subsequently to said geodesicconverter, for forming a low dimensional Euclidean representation ofsaid series of geodesic distances, said low dimensional Euclideanrepresentation providing a bending invariant representation of saidgeometric body suitable for matching with other geometric shapes.

The apparatus preferably comprises a subsampler located prior to saidgeodesic converter, configured to subsample said triangulated manifold,and to provide to said geodesic converter a subsampled triangulatedmanifold.

Preferably, said manifold comprises a plurality of vertices and whereinsaid subsampler is operable to select a first vertex and to iterativelyselect a next vertex having a largest geodesic distance from verticesalready selected, until a predetermined number of vertices has beenselected.

Preferably, said subsampler is operable to use the fast marching methodfor triangulated domains to calculate geodesic distances betweenvertices for said iterative selection.

Preferably, said geometric body is a face, having soft geometricregions, being regions of said face susceptible to short term geometricchanges, and hard geometric regions, being regions substantiallyinsusceptible to said short term geometric changes, said apparatuscomprising a preprocessor, located prior to said subsampler, forremoving said soft geometric regions from said face.

Preferably, said preprocessor is operable to identify said soft regionsby identification of an orientation point on said face.

Preferably, said orientation point is at least one of a nose tip, a pairof eyeball centers and a mouth center.

Preferably, said preprocessor is further operable to center said face.

Preferably, said preprocessor is further operable to crop said face.

Preferably, said preprocessor is operable to carry out removal of saidsoft regions by application of a geodesic mask.

Preferably, said geodesic converter is operable to use the fast marchingmethod for triangulated domains to calculate said geodesic distances.

The apparatus preferably comprises a triangulator for forming saidtriangulated manifold from scan data of a geometric body.

The apparatus may further be operable to embed said triangulatedmanifold into a space of higher than two dimensions, thereby to includeadditional information with said topographical information.

Preferably, said additional information is any one of a group comprisingtexture information, albedo information, grayscale information, andcolor information.

Preferably, said subsampler comprises an optimizer for allowing a userto select an optimum subsampling level by defining a trade-off betweencalculation complexity and representation accuracy.

Preferably, said multi-dimensional scalar is configured such that saidEuclidean representation comprises a predetermined number of eigenvaluesextractable to be used as co-ordinates in a feature space.

Preferably, said predetermined number of eigenvalues is at least three,and said feature space has a number of dimensions corresponding to saidpredetermined number.

According to a second aspect of the present invention there is providedapparatus for matching between geometric bodies based on 3-dimensionaldata comprising:

an input for receiving representations of geometric bodies as Euclideanrepresentations of sets of geodesic distances between sampled points ofa triangulated manifold, said Euclidean representations beingsubstantially bending invariant representations,

a distance calculator for calculating distances between respectivegeometric bodies based on said Euclidean representation and

a thresholder for thresholding a calculated distance to determine thepresence or absence of a match.

Preferably, said distance calculator comprises:

an eigenvalue extractor for extracting a predetermined number ofeigenvalues from said Euclidean representations, and

a plotter for plotting said predetermined number of eigenvalues as apoint on a feature space having a dimension for each of saidpredetermined number of Eigenvalues,

and wherein said thresholder is configured to be sensitive to clusteringwithin said feature space, thereby to determine said presence or absenceof said match. Preferably, said predetermined number is three.

Preferably, said Euclidean representation is based upon geodesicdistances between a subsampling of points of said triangulated manifold.

Preferably said geometric body is a face, having soft geometric regions,being regions susceptible to short term geometric change and hardgeometric regions, being regions substantially insusceptible to shortterm geometric changes, and wherein said Euclidean representation issubstantially limited to said hard geometric regions.

Preferably, said distance calculator is configured to use the Hausdorffmetric.

According to a third aspect of the present invention there is providedapparatus for obtaining 3-Dimensional data of geometric body formatching, and using said data to carry out matching between differentbodies, said apparatus comprising:

a three dimensional scanner for obtaining three-dimensionaltopographical data of said body,

a triangulator for receiving said three-dimensional topographical dataof said geometric body and forming said data into a triangulatedmanifold,

a geodesic converter, connected subsequently to said triangulator, forconverting said triangulated manifold into a series of geodesicdistances between pairs of points of said manifold,

a multi-dimensional scaler, connected subsequently to said geodesicconverter, for forming a low dimensional Euclidean representation ofsaid series of geodesic distances, said low dimensional Euclideanrepresentation providing a bending invariant representation of saidgeometric body,

a distance calculator, connected subsequently to said multi-dimensionalscaler, for calculating distances between geometric bodies based on saidEuclidean representation and

a thresholder, connected subsequently to said distance calculator, forthresholding a calculated distance to determine the presence or absenceof a match.

Preferably, said distance calculator comprises:

an eigenvalue extractor for extracting a predetermined number ofeigenvalues from said Euclidean representations, and

a plotter for plotting said predetermined number of Eigenvalues as apoint on a feature space having a dimension for each of saidpredetermined number of eigenvalues, and wherein said thresholder isconfigured to be sensitive to clustering within said feature space,thereby to determine said presence or absence of said match. Preferably,said predetermined number is three.

Alternatively, said predetermined number is greater than three.

The apparatus preferably comprises a subsampler located between saidtriangulator and said geodesic converter, configured to subsample saidtriangulated manifold, and to provide to said geodesic converter asubsampled triangulated manifold.

Preferably, said subsampler is operable to use geodesic distances inselecting points from said triangulated manifold to include in saidsubsampled triangulated manifold.

Preferably, said subsampler is configured to take an initial point andthen iteratively to select points by taking points furthest away interms of a geodesic distance from already selected points.

Preferably, said geometric body is a face, having soft geometric regionsand hard geometric regions, said apparatus comprising a preprocessor,located between said triangulator and said subsampler, for removing saidsoft geometric regions from said face.

Preferably, said geometric body is a face, having soft geometric regionsand hard geometric regions, said apparatus comprising a preprocessor,located between said triangulator and said geodesic converter, forremoving said soft geometric regions from said face.

Preferably, said preprocessor is operable to identify said soft regionsby identification of an orientation point on said face.

Preferably, said orientation point is a nose tip.

Preferably, said preprocessor is further operable to center said face.

Preferably, said preprocessor is further operable to crop said face.

Preferably, said preprocessor is operable to carry out removal of saidsoft regions by application of a geodesic mask.

Preferably, said geodesic converter is operable to use the fast marchingmethod for triangulated domains to calculate said geodesic distances.

Preferably, said subsampler comprises an optimizer for allowing a userto select an optimum subsampling level by defining a trade-off betweencalculation complexity and representation accuracy.

Preferably, said distance calculator is configured to use the Hausdorffmetric.

According to a fourth aspect of the present invention there is provideda method of image preprocessing of three-dimensional topographical datafor subsequent classification, the method comprising:

providing said three-dimensional topographical data as athree-dimensional triangulated manifold,

generating a matrix of geodesic distances to selected vertices of saidmanifold,

using multi-dimensional scaling to reduce said matrix to a canonicalrepresentation in a low-dimensional Euclidean space, thereby to providea representation suitable for subsequent classification.

The method may further comprise selecting said vertices for generatingsaid matrix of geodesic distances by a procedure comprising iterativelyselecting a next vertex having a largest geodesic distance from verticesalready selected, until a predetermined number of vertices has beenselected.

The method may further comprise determining geodesic distances for saidprocedure using the fast marching method for triangulated domains.

Preferably, said three-dimensional topographical data is arepresentation of a face, and further comprising cropping said face toexclude parts of said face being susceptible to short term changes,thereby to render said subsequent classification substantially invariantto said short term changes.

According to a fifth aspect of the present invention there is provided amethod of classifying images of three-dimensional bodies comprising:

obtaining representations of said three dimensional bodies as canonicalform representations derived from geodesic distances between selectedsample points taken from surfaces of said bodies,

from each representation deriving co-ordinates on a feature space, and

classifying said bodies according to clustering on said feature space.

Preferably, said deriving co-ordinates comprises deriving first meigenvalues from respective canonical form representations, where n is apredetermined number, and using said m eigenvalues to provide saidco-ordinates, said feature space being constructed to have n dimensions.

Preferably, n is at least three.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. The materials, methods, andexamples provided herein are illustrative only and not intended to belimiting.

Implementation of the method and system of the present inventioninvolves performing or completing selected tasks or steps manually,automatically, or as a combination thereof. Moreover, according toactual instrumentation arid equipment of preferred embodiments of themethod and system of the present invention, one or more steps could beimplemented by hardware or by software on any operating system of anyfirmware or a combination thereof. For example, as hardware, selectedsteps of the invention could be implemented as a chip or a circuit. Assoftware, selected steps of the invention could be implemented as aplurality of software instructions being executed by a computer usingany suitable operating system. In any case, selected steps of the methodand system of the invention could be described as being performed by adata processor, such as a computing platform for executing a pluralityof instructions.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, withreference to the accompanying drawings. With specific reference now tothe drawings in detail, it is stressed that the particulars shown are byway of example and for purposes of illustrative discussion of thepreferred embodiments of the present invention only, and are presentedin the cause of providing what is believed to be the most useful andreadily understood description of the principles and conceptual aspectsof the invention. In this regard, no attempt is made to show structuraldetails of the invention in more detail than is necessary for afundamental understanding of the invention, the description taken withthe drawings making apparent to those skilled in the art how the severalforms of the invention may be embodied in practice.

In the drawings:

FIG. 1 is a simplified block diagram showing a preferred embodiment of adevice for gathering 3D topographical data of a body and processing thedata into a canonical form representation for efficient matching,according to a first preferred embodiment of the present invention;

FIG. 2 is a simplified diagram of apparatus for receiving 3Dtopographical data in canonical form representation and for carrying outmatching or classification, according to a second preferred embodimentof the present invention;

FIG. 3 is a series of 2D representations of a single face, each takenunder different lighting conditions. The faces are clearly the same tothe human observer but are very difficult to match using conventionalimage analysis techniques;

FIG. 4 a is a simplified schematic diagram showing a first stage of 3Ddata gathering using depth code illumination;

FIG. 4 b is a simplified schematic diagram showing a second stage of 3Ddata gathering using depth code illumination;

FIG. 5 is a photograph showing a 3D scanner arrangement comprising avideo camera and a moving laser projector;

FIG. 6 is a simplified schematic diagram illustrating a photometricstereo acquisition scheme;

FIG. 7 is a triangulated manifold representation using data pointsgathered by 3D scanning of a face;

FIG. 8 is a simplified diagram showing the manifold of FIG. 7 aftersubsampling;

FIG. 9 is a simplified block diagram showing in greater detail thesubsampler of FIG. 1;

FIG. 10 is a simplified block diagram showing in greater detail thegeodesic converter of FIG. 1;

FIG. 11 is a simplified flow chart showing operation of a preferredembodiment of the present invention;

FIG. 12 is a simplified flow chart showing in greater detail thesubsampling stage of FIG. 11;

FIG. 13 is a database of six faces, used as the subject of the firstexperiment;

FIG. 14 shows one of the faces of FIG. 1, a variation of the face tosimulate change in expression, and a further variation to illustrate achange in a major feature;

FIG. 15 is a graph showing results of the first experiment plotted ontoa 3-dimensional feature space. Faces differing only by a change inexpression form a cluster on the feature space whereas faces differingby a change in a major feature are distant;

FIG. 16 illustrates nine different illuminations of a given face andtheir reconstruction, using photometric stereo, into a manifold bysolution of the Poisson equation;

FIG. 17 is a set of ten faces each reconstructed using least squaresfrom nine facial illuminations of a different subject for use in thesecond experiment;

FIG. 18 is a set of images of a single subject each with a differentpose, for use in experiment II;

FIG. 19 is a graph showing results of the second experiment, plotted ona 3-dimensional feature space. Different poses of the same subject formclusters;

FIG. 20 is a schematic diagram illustrating an alignment process of twofacial surfaces;

FIG. 21 is a graph showing the results of the third experiment withoutalignment, plotted on a three-dimensional feature space. No recognizableclusters appear;

FIG. 22 is a graph showing the results of the third experiment carriedout with optimal alignment, plotted on a three-dimensional featurespace. Clustering is present but indistinct;

FIG. 23 is a set of three faces from a further database, the faces usedas subjects for the fourth experiment;

FIG. 24 is a set of three poses of a face of FIG. 23, also for use inthe fourth experiment;

FIG. 25 shows a texture map of a face before application of a mask aspart of preprocessing;

FIG. 26 shows the texture map of FIG. 25 after application of a geodesicmask;

FIG. 27 shows a depth map of a face before application of a mask as apart of preprocessing;

FIG. 28 shows the depth map of FIG. 27 after application of a geodesicmask;

FIG. 29 shows a triangulated manifold as obtained directly from 3Dtopographical image data;

FIG. 30 shows the manifold of FIG. 29 following subsampling; and

FIG. 31 shows the manifold of FIG. 29 reduced to canonical form.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present embodiments show a face recognition approach based on 3Dgeometric or topographical information. Given a 3D facial surface, ageometric bending-invariant canonical form can be obtained by samplingthe surface, computing the geodesic distances between points on it(using the Fast Marching method) and applying Multidimensional scaling(MDS). MDS allows representation of the surface in a low-dimensionalEuclidean space, the bending invariant canonical form, and the canonicalform is a representation which can be used for classification in generaland matching in particular.

Facial recognition using 3D geometry may be used on its own or toaugment conventional 2D imaging. As the 3D geometry is independent ofviewpoint and lighting conditions, accurate face classification can beachieved.

Before explaining at least one embodiment of the invention in detail, itis to be understood that the invention is not limited in its applicationto the details of constriction and the arrangement of the components setforth in the following description or illustrated in the drawings. Theinvention is capable of other embodiments or of being practiced orcarried out in various ways. Also, it is to be understood that thephraseology and terminology employed herein is for the purpose ofdescription and should not be regarded as limiting.

Referring now to the drawings, FIG. 1 is a simplified diagram showingapparatus for obtaining 3-Dimensional data of a geometric body forclassification, including matching, according to a first preferredembodiment of the present invention. The preferred embodiments relatespecifically to matching of faces but the skilled person will be awarethat the principles of the present invention ire applicable to anygeometric body having a three-dimensional structure.

Apparatus 10 comprises a three dimensional scanner 12 for obtainingthree-dimensional topographical data of the body. Several types ofscanner are described hereinbelow together with brief discussions of howto process data therefrom in some of the cases.

Data from the three-dimensional scanner 12 is passed to a triangulator14. The triangulator may perform triangulation on the data received fromthe scanner in order to generate a three-dimensional triangulatedmanifold to represent the topological features of the body. The exactoperation of the triangulator to derive the manifold may vary dependingon the way in which the 3D information is gathered. In certain cases themanifold may be formed directly from the gathered data without the needfor any intervening processing stage. The manifold preferably representsall of the three-dimensional topology of the body and therefore is intheory sufficient for allowing matching. However, in practice: directcomparisons using the triangulated manifold have a number ofdisadvantages, as will be demonstrated in experiment 3 hereinbelow. Theyrequires a large amount of calculation. Matching does not distinguishreliably between different faces. Moreover matching generally fails whenthe same face has a different expression and matching is unreliable evenwhen the same face is posed at a different angle.

Embodiments of the present invention therefore preferably include fouradditional processing stages, the first of which is a preprocessor 16.Preprocessor 16 takes a reference point in order to determine anorientation around the manifold. A reference point which is relativelyeasy to find automatically from a manifold of a face is the tip of thenose. Other possible reference points include centers of eyeballs andthe center of the mouth. Once the preprocessor has found the tip of thenose it is able to orientate itself with respect to the rest of the faceand then parts of the face whose geometry is particularly susceptible toexpressions, hereinafter referred to as soft regions, can be ignored.Parts of the face that are invariant with change of expression and thelike, hereinafter hard regions, can be retained or even emphasized. Aswill be explained in greater detail below, the definition of softregions is not fixed. For some methods and in some circumstances softregions to be excluded may include all of the lower region of the facearound the mouth. In other cases less drastic exclusions may beconsidered. In one embodiment, described in greater detail below, softregions are removed using a geodesic mask. The mask may be appliedseparately to a texture map of the face and a depth map of the face.

Following the preprocessor is a subsampler 18. The subsampler 18 takesthe preprocessed manifold and removes points so as to produce a lesswell defined manifold, but one which still defines the essentialgeometry of the face it is desired to match. In preferred embodiments,the user is able to select the number of points to trade off betweenaccurate matching—a large number of points—and faster processing—asmaller number of points. As will be discussed in greater detail below,a preferred embodiment of the sub-sampler uses the Voronoi subsamplingtechnique which begins at an initial point or vertex on the manifold andthen adds the point or vertex having the greatest distance therefrom.The procedure is repeated iteratively until the selected number ofpoints are included. Preferably the technique uses geodesic distances,which may be obtained using the fast marching method for thetriangulated domain (FMM-TD), as described below.

Following the subsampler is a geodesic converter 20. The geodesicconverter 20 receives the list of points of the subsampled manifold andcalculates a vector for each pair of points. The vectors are expressedas geodesic distances, and the fast marching algorithm for thetriangulated domain is again used to obtain the geodesic distances in anefficient manner as possible.

Following the geodesic converter is a multi-dimensional scaler 22, whichtakes the matrix of the geodesic distances calculated by the geodesicconverter 20, referred to below as the distance matrix, and forms a lowdimensional Euclidean representation of the series of geodesicdistances, using multi-dimensional scaling. Multi-dimensional scaling isdiscussed in detail below. The low dimensional Euclidean representationprovides a bending invariant representation of the geometric body, aswill be explained in the discussion of multi-dimensional scaling below.The use of such a bending invariant representation ensures that thematching process is not fooled by, for example, scans of the head atdifferent angles.

The output 24 of the multi-dimensional scalar is a representation of the3D face in terms of Euclidean distances between surface points, referredto hereinbelow as the canonical form representation.

Reference is now made to FIG. 2, which is a simplified diagram showing amatching apparatus for matching two faces using the canonical formoutput as described above. The matcher 30 may be a continuation of theapparatus 10 or may be supplied as a separate unit. The matcher 30comprises a distance calculator 32, which takes as input two canonicalform representations 34 and 36, and calculates a distance therebetween.The distance calculation may use any suitable method for comparison ofthe canonical forms for each of the faces to be matched. Astraightforward approach is to measure a distance between two sets ofpoints, using, for example the Hausdorff metric. However, the Hausdorffmetric based method is computationally extensive.

An alternative approach, used in the present embodiments, takes thefirst m eigenvalues obtained from the MDS procedure to providecoordinates in a low-dimensional feature space. Although the dominanteigenvalues do not describe the canonical form entirely, it isreasonable that similar faces have similar eigenvalues (and thus formclusters in the feature space). A distance is calculated between thegeometric bodies, or, as will be described below, plotted on a graph ofthe feature space and a thresholder 38, which is connected subsequentlyto the distance calculator, thresholds the calculated distance todetermine the presence or absence of a cluster in the feature space, thecluster indicating a match. In the embodiments described in detailherein, the first three Eigenvalues are taken and are plotted in a threedimensional feature space.

Reference is now made to FIG. 3, which shows a series of threetwo-dimensional images. It will be apparent to any human observer thatthe three images are of the same face, however conventional automaticmatching techniques generally find very large distances between thethree images and consequently fail to match them. Thus two-dimensionalfacial matching is prone to errors introduced by simple variables suchas lighting direction. In fact, areas of the face that have highreflectivity, such as the eyes, can change substantially for very minorchanges in lighting.

Returning now to FIG. 1, and as mentioned above, apparatus 10 preferablycomprises a three-dimensional scanner 12. The face recognition describedin the present embodiments treats faces as three-dimensional surfaces.It is therefore first of all necessary to obtain the facial surface ofthe subject that it is desired to recognize. Below is a short overviewof currently available range finding techniques that are able to scan aface and generate three-dimensional data.

Laser Range Camera (Zeam)

Reference is now made to FIGS. 4 a and 4 b, which are simplifieddiagrams showing successive stages of operation of a laser range camerausing depth encoding illumination. Currently, the fastest and mostaccurate, but at the same time most expensive, range cameras are thosethat are based on depth-encoding illumination.

Depth-encoded illumination is based upon the generation of a wall oflight 40 which moves along the field of view. As the light hits theobjects 42 in the scene, it is reflected back towards the camera 44. Dueto the finite speed of the light, the reflection carries an imprint ofthe object depth which may be decoded using the appropriate electronics.The illumination is typically generated using IR laser diodes.

A 3D acquisition equipment of this type is known from WO Patent01/18563, Mar. 15, 2001, the contents of which are hereby incorporatedby reference. Such a technology allows real-time data acquisition atabout 30 fps frame rate. Depth resolution can be greatly improved byaveraging frames in time. Typical prices range between US$2K-50K.

3D Scanner

A slower and cheaper version of a 3D scanner is based upon J. -Y.Bouguet and P. Perona, “3D photography on your desk”, in Proc. of theInt. Conf. on Computer Vision, Bombay, India, January 1998. The scanneris based on a lamp and a pencil casting a line of shadow on a desk, andan implementation by Zigelman and Kimmel uses a narrow laser beaminstead of shadow, see G. Zigelman and R. Kimmel, “Fast 3D laserscanner”, Dept. of Computer Science, Technion—Israel Institute ofTechnology, Tech. Rep. CIS-2000-07, 2000. A typical device is shown inFIG. 5 and comprises a video camera 46 and moving laser projector 48.Using the device of FIG. 5, depth reconstruction is performed byanalyzing the laser beam deformation as it illuminates the object. Arelated approach is discussed below under the heading “structuredlight”.

Such a scanner can be constructed from cheap materials having a cost ofbetween US$ 50-200. Typical scan speeds for faces achievable with suchdevices are within the range 15-30 sec, and the low scan speed limitsthe approach to laboratory and like controlled conditions.

Structured Light

The idea of structured light relates to the projecting of a knownpattern (e.g. parallel stripes) on to an object, and subsequentlycapturing depth information from the pattern deformation. Furtherdetails are available from C. Rocchini, P. Cignoni, C. Montani, P. Pingiand R. Scopigno, A low cost 3D scanner based on structured light,EUROGRAPHICS 2001, A. Chalmers and T.-M. Rhyne (Guest Editors), Volume20 (2001), Number 3, the contents of which are hereby incorporated byreference.

Typical data acquisition setup includes a CCD camera and a projector andis relatively inexpensive with current costs being in the region ofUS$1K-5K. Such a data acquisition device was used by Beumier and Acheroyfor 3D face recognition, and reference is made to C. Beumier and M. P.Acheroy, Automatic Face Identification, Applications of Digital ImageProcessing XVIII, SPIE, vol. 2564, pp. 311-323, July 1995, and to C.Beumier and M. P. Acheroy, Automatic Face Authentication from 3DSurface, British Machine Vision Conference BMVC 98, University ofSouthampton UK, 14-17 Sep. 1998, pp 449-458, 1998, the contents of whichare hereby incorporated by reference.

The disadvantage of the structured light method is the need forcontrolled illumination conditions, again rendering the methodinapplicable to many real life applications.

Photometric Stereo

An alternative way of facial surface acquisition, which does not requireany dedicated hardware, is surface reconstruction from photometricstereo. Photometric stereo requires the acquiring of several images ofthe same subject in different illumination conditions and extracting the3D geometry from the images by assuming a Lambertian reflection model. Asurface is said to exhibit Lambertian reflection if it behaves as a dullor matte surface. That is to say, for incident light from any direction,the reflection is entirely diffuse, meaning that light is reflected inequal intensity in all directions. For any given surface, the brightnessdepends only on the angle between the illumination source and thesurface normal.

Reference is now made to FIG. 6, which is a simplified diagram showing afacial surface 50, represented as a function, viewed from a givenposition 52 on the z-axis. The object is illuminated by a source ofparallel rays 54 directed along I¹.

In the following, Lambertian reflection is assumed and the observedpicture is thus given byI ^(i)(x,y)=ρ(x,y)

n(x,y),I ¹

,  (1)

where p(x,y) is the object albedo or reflective power, and n(x,y) is thenormal to the object surface, expressed as $\begin{matrix}{{n( {x,y} )} = {\frac{\lbrack {{- {z_{x}( {x,y} )}},{- {z_{y}( {x,y} )}},1} \rbrack}{\sqrt{1 + {{\nabla{z( {x,y} )}}}_{2}^{2}}}.}} & (2)\end{matrix}$

Using matrix-vector notation, equation (1) can be rewritten asI(x,y)=Lv  (3)

where $\begin{matrix}{{L = {- \begin{bmatrix}l_{1}^{1} & l_{2}^{1} & l_{3}^{1} \\\vdots & \vdots & \vdots \\l_{1}^{N} & l_{2}^{N} & l_{3}^{N}\end{bmatrix}}};{{I( {x,y} )} = \begin{bmatrix}{I^{1}( {x,y} )} \\\vdots \\{I^{N}( {x,y} )}\end{bmatrix}}} & (4) \\{and} & \quad \\{{v_{1} = {z_{x}v_{3}}};{v_{2} = {z_{y}v_{3}}};{v_{3} = {\frac{\rho( {x,y} )}{\sqrt{1 + {{\nabla z}}_{2}^{2}}}.}}} & (5)\end{matrix}$

Given at least 3 linearly independent illuminations {I^(i)}_(i+I) ^(N),and the corresponding observations {I^(i)}_(i=I) ^(N), one canreconstruct the values of ∇z by pointwise least-squares solutionv=L ^(T) I(x,y)  (6)

where L^(T)=(L^(T)L)⁻¹L^(T) denotes the Moore-Penrose pseudoinverse. Itis noted that the Moore-Penrose pseudoinverse is a substitute for theinverse matrix, and is useful in circumstances in which a standardinverse matrix does not exist. A function for obtaining theMoore-Penrose pseudoinverse of a matrix is provided for example as astandard feature in Matlab.

Having the gradient ∇z, the surface 50 can be reconstructed byminimization of the following function: $\begin{matrix}{\overset{\_}{z} = {\underset{\overset{\_}{z}}{\arg\quad\min}{\int{\int{\lbrack {( {{\overset{\_}{z}}_{x} - z_{x}} )^{2} + ( {{\overset{\_}{z}}_{y} - z_{y}} )^{2}} \rbrack{\mathbb{d}x}{\mathbb{d}y}}}}}} & (7)\end{matrix}$

The Euler-Lagrange conditions of equation (7), in turn, allow rewritingof the Poisson equation thus,{overscore (z)} _(xx) +{overscore (z)} _(yy) ={overscore (z)} _(xx)+{overscore (z)} _(yy)  (8)

[I] the solution of which yields a reconstructed surface {overscore(z)}. See R. Kimmel, Numerical geometry of images, Lecture notes.

One of the obvious problems in the surface reconstruction fromphotometric stereo approach arises from deviations from the Lambertianmodel in real faces. In face recognition applications, some facefeatures (such as eyes, hair, beard etc.) have a strongly non-Lambertiannature.

To reduce the irregularities in the reconstructed surface, one can add apenalty term on surface non-smoothness to the least-squares solution,for example to give a total variation: $\begin{matrix}{\overset{\sim}{z} = {{\underset{\overset{\_}{z}}{\arg\quad\min}{\int{\int{\lbrack {( {{\overset{\_}{z}}_{x} - z_{x}} )^{2} + ( {{\overset{\_}{z}}_{y} - z_{y}} )^{2}} \rbrack{\mathbb{d}x}{\mathbb{d}y}}}}} + {\lambda{\int{\int{{{\nabla z}}_{2}{\mathbb{d}x}{\mathbb{d}y}}}}}}} & ( {8a} )\end{matrix}$

Geometric Face Recognition Scheme

The face recognition scheme of the present embodiments is based on thegeometric features of the three-dimensional facial surface. Thegeometric features may be obtained either directly from a 3D scan andincorporated as points or vertices into a triangulated manifold of thekind shown in FIG. 7, or the vertices may be derived from photometricstereo and then arranged to form the triangulated manifold.

Preliminary processing, such as centering and cropping, is preferablyperformed on the manifold prior to recognition. Centering and croppingactions can be carried out by simple pattern matching, which may use theeyes, or the tip of the nose 60, as the most recognizable feature of thehuman face. The facial contour may also be extracted in order to limitprocessing to the surface belonging to the face itself, thus to excludeforeground items such as spectacles and the like. Such tasks arepreferably performed by the preprocessor 16 referred to above.

Preprocessing preferably emphasizes those sections of the face lesssusceptible to alteration, the so-called hard regions, including theupper outlines of the eye sockets, the areas surrounding one'scheekbones, and the sides of the mouth. On the other hand, sections,which can be easily changed (e.g. hair), the soft regions, arepreferably excluded from the recognition process. As will be discussedbelow, regions that change significantly according to expression canalso be included as soft regions and excluded, and in certainembodiments processing may be limited just to the upper part of theface.

Following treatment by the preprocessor, the manifold typically stillcomprises too many points for efficient processing and more points thanare really necessary to convey the underlying facial geometry. Hencesubsampling is carried out using subsampler 18 in order to form thesubsampled manifold of FIG. 8.

Reference is now made to FIG. 9, which is a simplified block diagramshowing the subsampler 18 in greater detail. The subsampler comprises aninitializer 70, and a Voronoi sampler 72, and takes as inputs thepreprocessed full manifold and a desired number of points, or any otheruser-friendly way of defining a trade-off between accuracy andcomputational efficiency.

In the subsampling procedure, a subset of n uniformly distributedvertices is selected from the triangulated surface within the facialcontour The sub-sampling is performed using the iterative Voronoisampling procedure, where on each iteration a vertex with the largestgeodesic distance from the already selected ones is selected. Theprocedure is initialized by a constant vertex selected by theinitializer and the geodesic distances needed are preferably computedusing FMM-TD, the mathematics of which are discussed in greater detailbelow. For further information on Voronoi sampling, reference is made toCISM Journal ACSGC Vol. 45 No. 1. Spring 1991 pp 65-80, Problems withhandling spatial data, the Voronoi approach, Christopher M. Gold. FMM-TDis discussed in greater detail below.

Reference is now made to FIG. 10, which is a simplified block diagramshowing in greater detail the geodesic converter 20 of FIG. 1. In thegeodesic converter 20, sets of points 80 are received from thesubsampler 18. An n×n distance matrix of geodesic distances is thencreated by applying FMM-TD from each of the n selected vertices to eachother vertex. One of the principles ensuring the low constitutionalcomplexity of FMM-TD is the fact that already calculated distances fromthe Voronoi sampling can be reused in subsequent computations such ascomputing the geodesic distance matrix.

Using the n×n distance matrix, multi-dimensional scaling (MDS) isapplied to produce a dissimilarity matrix, a map of similarities ordissimilarities that provides a canonical representation of the face ina low-dimensional Euclidean space. Since isometric transformations donot affect the geodesic distances, it may be expected that a faceundergoing such a transformation yields the same canonical form as thatof the original face. Studies carried out by the inventors show thatindeed slight non-isometric transformations produce small butinsignificant deviations from the original canonical form. MDS isdiscussed in greater detail below.

The last stage of the facial matching operation consists of matching orclassification, i.e. comparison of the canonical forms. Astraightforward approach is to measure a distance between two sets ofpoints, using, for example, the Hausdorff metric. However, the Hausdorffmetric based method is computationally extensive and should be thereforeavoided.

An alternative approach, used in the present embodiments, takes thefirst m eigenvalues obtained from the MDS procedure to providecoordinates in a low-dimensional feature space. Although the dominanteigenvalues do not describe the canonical form entirely, it isreasonable that similar faces have similar eigenvalues (and thus formclusters in the feature space). Experiments, cited below, show that thedescribed comparison is sufficiently accurate and has low computationalcomplexity. In the present embodiments the first three eigenvalues aretaken and plotted onto a three-dimensional feature space.

Fast Marching on Triangulated Manifolds

Face recognition according to the present embodiments uses geodesicdistances between points on the facial surface, both for subsampling andfor creating a distance matrix from the subsampled points. Initially,one computes distances between pairs of points on a triangulatedmanifold representing the facial surface, and then an efficientnumerical method is required to obtain geodesic distances therefrom.

A method known as The Fast Marching Method (FMM), is disclosed by J. A.Sethian, A fast marching level set method for monotonically advancingfronts, Proc. Nat. Acad. Sci., 93, 4, 1996, the contents of which arehereby incorporated by reference. The fast marching method wassubsequently extended to triangulated domains (FMM-TD) as disclosed inR. Kimmel and J. A. Sethian, Computing geodesic paths on manifolds, thecontents of which are likewise incorporated by reference. FMM-TD is anefficient numerical method to compute a first-order approximation of thegeodesic distances.

Given a set of source points {s₁} on a manifold such as that of FIG. 7,the distance map T(x,y) from these points to other points on themanifold is obtained as the solution of the Eikonal equation∥∇T∥=1; T(s ₁)=0  (9)

FMM-TD allows computing of the distance map with O(N log N) complexity,where N is the number of points on the manifold.

When the face geometry is obtained from photometric stereo additionalefficiency can be achieved as follows. There is no actual need tocompute the surface itself from equation (8) above. It is sufficient tofind the gradient ∇z, and use its values to construct the metric used bythe FMM.

Multidimensional Scaling

Multidimensional scaling (MDS) is a method that maps measuredsimilarities or dissimilarities among a set of objects into arepresentation of the pattern of proximities in a low-dimensionalEuclidean space, and in this context, reference is made to G. Zigelman,R. Kimmel, and N. Kiryati, Texture mapping using surface flattening viamulti-dimensional scaling, Accepted to IEEE Trans. on Visualization andComputer Graphics, 2001, and R. Grossmann, N. Kiryati, and R. Kimmel.Computational surface flattening: A voxel-based approach. Accepted toIEEE Trans. on PAMI, 2001, the contents of both of these documentshereby being incorporated herein by reference.

Given a set of n objects, their mutual similarities {d_(ij)}_(i,j−1)^(n) and the desired dimensionality m, MDS finds a set of vectors inm-dimensional space (each vector corresponding to an object) such thatthe matrix of Euclidean distances among them corresponds as closely aspossible to a function of the input matrix D according to a certaincriterion function.

In the present embodiments, proximity values are obtained by measuringthe geodesic distances between points on the facial surface using theFMM-TD method. A. Elad and R. Kimmel, Bending invariant representationsfor surfaces, Proc. of CVPR′01 Hawaii, December 2001, the contents ofwhich are hereby incorporated herein by reference, showed that applyingMDS to geodesic distances on a manifold produces a bending-invariantcanonical form.

There exists a variety of different algorithms for solving the MDSproblem; in each of them a trade-off between computational complexityand algorithm accuracy is made. The present embodiments make use of theclassical scaling algorithm introduced by Young et al. however theskilled person will appreciate the applicability of other methods.

Classical scaling finds the coordinates x_(i) of n points in ak-dimensional Euclidean space, given their mutual distances{d_(ij)}_(i,j−1) ^(n). The Euclidean distance between the points i and jis expressed by $\begin{matrix}{d_{ij}^{2} = {{{x_{i} - x_{j}}}_{2}^{2} = {( {x_{i} - x_{j}} )^{T}{( {x_{i} - x_{j}} ).}}}} & (10)\end{matrix}$

Given the squared-distance matrix D with elements as in (10) one canconstruct the inner-product matrix $\begin{matrix}{B = {{{- \frac{1}{2}}( {I - {\frac{1}{n}11^{T}}} ){D( {I - {\frac{1}{n}11^{T}}} )}} = {XX}^{T}}} & (11)\end{matrix}$

where I=[1, . . . , l]^(T) and X=[x₁, . . . , x_(n)]^(T). B is apositive semi-definite matrix with at most k positive eigenvalues, whichcan be expressed asB=UAU^(T).  (12)

The coordinate matrix X is therefore obtained by computing the squareroot matrix of BX=UA^(1/2).  (13)

It is noted that, from a statistical point of view, the above-describedapproach is equivalent to principal component analysis (PCA), whichfinds an orthogonal basis that maximizes the variance of the given nvectors projected to the basis vectors.

Empirical observations show that three dimensions usually suffice forrepresentation of most of the geometric structure contained in thegeodesic distances measured on a smooth surface.

Reference is now made to FIG. 11, which is a simplified flow chartshowing the process flow of embodiments of the present invention, andshowing how the various procedures and functions described above maywork together. In a first stage S1, 3D data is collated from the objectit is desired to match. Collation may use any of the scanning methodsdescribed above or any other suitable method of gathering 3D topologicalinformation of a body. In an optional stage S2, albedo or reflectivitypower information may be gathered. One of the possibilities ofincorporating albedo information is by embedding the two-dimensionalface manifold into a 4D or 6D space and measuring distances on themanifold using a combined metric, thus the 4^(th) coordinate in the4D-embedding and the 4^(th-)6^(th) coordinates in the 6D-embeddingrepresent the gray level or the RGB channels of the albedo information,respectively.

The data gathered is then presented in a stage S3 as a series of pointson a triangulated manifold such as that of FIG. 6. The manifold may beconstructed directly from the 3D data or may require auxiliarycalculations, depending on the data gathering method used.

Stage S4 is a preprocessing stage. The preprocessing stage removessoft—that is often changing regions of the face geometry and retainshard regions, that is those regions that remain constant. Preprocessingmay also include cropping the image and like operations. Preprocessingpreferably involves locating a reference point on the face and thenusing the general facial form to determine the locations of the softregions that are to be ignored.

Stage S5 is a subsampling stage. The preprocessed manifold is reduced byselecting only those points which are most crucial for retaininggeometric information. As discussed above, efficient techniques such asVoronoi sampling may be used. Stage S5 is shown in greater detail inFIG. 12 in which a stage S5.1 comprises input of the preprocessedmanifold. A selection stage S5.2 allows a user to set a desired numberof points for the subsampling in a trade-off between accuracy andcomputational complexity. Finally a stage 5.3 involves Voronoi sampling,which looks for the minimum number of points having the maximumtopological information. As discussed above, an initial point is takenand then a farthest point therefrom is added until the desired number ofpoints is reached. Geodesic distances are used to determine furthestpoints and FMM-TD can be used to find the geodesic distances.

In a stage S6, a distance matrix is computed of geodesic distancesbetween each subsampled vertex and each other point. As explained above,FMM-TD is preferably used as an efficient method of computing geodesicdistances.

In a stage S7, MDS is used to measure dissimilarity in the distancematrix and thereby express the geodesic distances in a low dimensionalEuclidean representation. Finally in a stage S8, the face as representedin low dimensional Euclidean representation can be compared with otherfaces by various distance calculation techniques. As discussed above, apreferred method uses the first m eigenvalues obtained from the MDSprocedure to provide coordinates in a low-dimensional feature space. Inthe present embodiments the first three eigenvalues are taken andplotted on a three-dimensional feature space. Although the dominanteigenvalues do not describe the canonical form entirely, similar facesnevertheless form recognizable clusters in the feature space and thusallow matching.

Experimental Results

Four experiments were performed in order to evaluate the approach andvalidate the present embodiments. Three databases were used:

I. A database ¹ of 3D facial surfaces obtained by a simple 3D scanner ofthe kind described above and shown in FIG. 13. The database consisted of6 subjects a . . . f including one artificial face (subject b) and 5human faces.

II. The Yale Face Database B ². The database consists of high-resolutiongrayscale images of 10 subjects of both Caucasian and Asian type, takenin controlled illumination conditions. Each subject was depicted inseveral poses (slight head rotations). 3D surface reconstruction wascarried out using photometric stereo.

III. A Database of high-resolution facial surfaces and textures of humansubjects with different facial expressions, obtained using the 3Dscanner.

Scanned Surfaces with Artificial Facial Expressions

The first experiment was performed on database I (FIG. 13) and wasintended to test algorithm sensitivity to artificial facial expressions.

Reference is now made to FIG. 14 which shows how one of the faces ofFIG. 13 was deformed to simulate facial expressions. Face a isundeformed face d of FIG. 13 and faces b and c are differentdeformations applied thereto. In b, the forehead, check and chins ofsubject d were deformed, simulating different facial expressions of thesame subject. In c, a feature, the nose, was substituted with that of adifferent face. The latter was applied to test algorithm sensitivity tomajor feature alteration It is emphasized that these deformations werenot isometric.

The surfaces were scaled to the size 60×95 and downsampled to 400 pointsusing Voronoi sampling. Afterwards, the geodesic distance from eachpoint was computed using FMM-TD and multidimensional scaling was appliedto the distance matrix.

Reference is now made to FIG. 15, which is a graph of a 3-dimensionalfeature space for matching. More specifically, the graph of FIG. 15depicts the feature-space representation on which the first 3eigenvalues obtained by MDS may be plotted. Each point in the featurespace corresponds to one of the faces in FIGS. 13 and 14, viz A (dot), B(circle), C (cross), D (pentagram), E (triangle) and F (hexagram). Itcan be seen that face deformations, though not purely isometric, produceclosely located points that form clusters. On the other hand, facesobtained by feature substitution are far from the original face ofsubject D.

Photometric Stereo

The second experiment was performed on database II. The aims of theexperiment were to determine the feasibility of using photometric stereofor 3D surface acquisition and the sensitivity of the algorithm to headrotations.

It appears that faithful surface reconstruction from photometric stereodemands a large number of different illumination directions. The reasonis primarily due to deviations in practice from the Lambertian model(especially notable in forehead, hair and eyes) and slight subjectdisplacement during image acquisition (e.g. eye blinking). Reference isnow made to FIG. 16, which shows nine different illuminations of thesame face and a least squares surface reconstruction made by surfacereconstruction from all nine of the images. Nine is simply a numberfound empirically to be an efficient number. Smaller numbers of imagesresulted in less accurate surfaces and therefore less accuraterecognition.

Reference is now made to FIG. 17, which depicts a series of ten facialsurfaces a-j each reconstructed from photometric stereo from nine imagesof respective faces. It is noted that relatively high noise appears tobe inevitable when reconstructing 3D geometry from photometric stereo.

Reference is now made to FIG. 18, which shows subject G of FIG. 17 takenat several poses. For some subjects, several poses were taken to testthe recognition sensitivity to small head rotations. Since the algorithmis bending invariant, the influence of rotation (an isometrictransformation) should be negligible, and all the possible deviationsfound seem mainly to be due to inaccurate surface reconstruction.

In the experiment, faces were manually centered according to the eye andlip locations and processing was performed in the region bounded by thefacial contour. The surfaces were scaled to the size 56×80 and sampledat 400 points using Voronoi sampling. Afterwards, the distance from eachpoint was computed using fast marching, and multidimensional sealing wasapplied to the distance matrix.

Reference is now made to FIG. 19, which is a simplified graph depictingthe feature-space representation using the first 3 eigenvalues obtainedby singular value decomposition of the faces of FIG. 17, some of them inseveral poses as exemplified by FIG. 18. A (dot), B (circle), C (cross),D (plus), E (star), F (pentagram), G (hexagram), H (diamond), I (square)and J (triangle). Different poses of subjects A, B, D, E and G formclusters. It is clear from the results of the second experiment thatdifferent poses result in relatively narrow clusters.

Surface Matching

The third experiment was performed on database I, the aim of theexperiment being to compare the geometric approach to a straightforwardsurface comparison.

The experiment uses a simple L₂-norm-based measure for surfacesimilarity. Reference is now made to FIG. 20, which shows two pairs ofsurfaces 90 and 92. Each surface has a plane 94, 94′ defined by the twoeyeball centers and the center of the mouth. Pair of surfaces 90 are notaligned and pair of surfaces 92 are the same as those in 90 but afteralignment. In the experiment, the surfaces were aligned in two differentways:

(a) as shown in FIG. 20, so that the planes 94, 94′ formed by the twoeyeball centers and the center of the mouth in the two surfacescoincided, and

(b) by finding the Euclidean transformationRx|b  (14)

in which R is a rotation matrix, and b is a translation vector, so as tominimize the sum of Euclidean distances between the points on the twosurfaces, that is the rotation is selected that minimizes the surfacedissimilarity (in this case, the approximated volume in the gap betweentwo aligned surfaces). Optimal alignment was found by the gradientdescent algorithm with numeric gradient computation.

Once an optimal alignment is found, the proximity of surfaces iscomputed by integrating the volume of the gap between the two surfaces.Such an approach follows the global surface matching idea disclosed inC. Beumier and M. P. Acheroy, Automatic Face Authentication from 3DSurface, British Machine Vision Conference BMVC 98, University ofSouthampton UK, 14-17 Sep. 1998, pp 449-458.”, 1998, the contents ofwhich are hereby incorporated by reference.

For visualization purposes and for displaying the subjects as points in3D space a matrix D is formed of the distances between the surfacesaccording to $\begin{matrix}{d_{ij} = {\sum\limits_{k - 1}^{N}{{x_{k}^{i} - x_{k}^{j}}}_{2}}} & (15)\end{matrix}$

where {x_(k) ^(i)}_(i−1) ^(N) are the N points forming the i-th surface,and applying MDS to D. The first three dominant eigenvalues were takenin order to obtain the representation of the proximity pattern in athree-dimensional Euclidean space.

Reference is now made to FIGS. 21 and 22, which are two graphs summingup the results of this third experiment. It is apparent that thestraightforward comparison of manifolds after alignment is sensitive tofacial expressions and fails to distinguish between differentexpressions of the same subject on the one hand and different subjectson the other hand.

More particularly, FIGS. 21 and 22 show that unlike the geodesicdistances, L₂ distances between the surfaces do not distinguish betweenfaces, even using an optimal surface alignment. Different expressionsand head poses of the same subject do not form recognizable clusters.The reason is the fact that the straightforward comparison does not takeinto consideration the geometry of the facial surfaces.

FIG. 21 shows the results of an L₂-based algorithm for alignmentaccording to 3 points for the set of faces of FIG. 17 and some of thefaces showing different expressions: A (dot), B (circle), C (cross), D(plus): E (star), F (pentagram), G (hexagram), H (diamond), I (square)and J (triangle). No clusters are recognizable.

FIG. 22 uses an L₂-based algorithm with optimal alignment for the sameset of faces: A (dot), B (circle), C (cross), D (plus), E (star), F(pentagram), G (hexagram), H (diamond), I (square) and J (triangle).Again, no distinct clusters appear.

Human Subjects with Real Expressions

A fourth experiment was performed, this time using database III. Themain goal was testing the algorithm in field conditions, on subjectswith real facial expressions. Three subjects from database III are shownin FIG. 23, and FIG. 24 shows a single subject with three differentexpressions.

Reference is now made to FIGS. 25-28 which show preprocessing of thefaces to extract only features that are relevant. The faces wereprocessed in order to extract the relevant features only. A geodesicmask was applied both to the texture and the depth maps. The mask wasobtained by computing a ball of constant radius—in the geodesicsense—around the tip of the nose. In the experiment the nose locationwas found manually. FIGS. 25 and 26 are texture maps, and FIGS. 27 and28 are depth maps. FIGS. 25 and 27 show texture and depth mapsrespectively before application of the mask and FIGS. 26 and 28 showtexture and depth maps respectively after application of the mask.

Reference is now made to FIGS. 29, 30 and 31, which respectively showmesh processing stages of the present embodiments. FIG. 29 shows thehigh-resolution triangulated manifold, FIG. 30 shows the downsampledversion of the manifold and FIG. 31 shows the surface in canonical form.

The minimal mean squared distance measure yielded much better resultsapplied to the canonical surface rather than to the original manifold.Currently, the minimal mean squared distance measure appears to becomputationally expensive. Canonical form comparison, based on centralmoments which are much simpler to compute, yielded a bit less accurateresults than the minimal squared distance, but still provided a goodmeasure for face similarity.

The classical scaling algorithm was used for creating the canonicalforms. The present inventors have seen no significant improvement overthe classical scaling algorithm in using the least-square MDS algorithm.250-500 subsamples have been found to give sufficient accuracy whilekeeping the computational complexity relatively low.

It is noted that the present embodiments yield poor results whenconsidering strong non-isometric deformations of the facial surface suchas inflated cheeks or open mouth. Limiting the region of interest to theupper part of the face, being a hard region and containing substantiallyrigid features, on the preprocessing stage yields significantly betterresults under the above conditions.

It is appreciated that certain features of the invention, which are, forclarity, described in the context of separate embodiments, may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention, which are, for brevity, described in thecontext of a single embodiment, may also be provided separately or inany suitable subcombination.

Although the invention has been described in conjunction with specificembodiments thereof, it is evident that many alternatives, modificationsand variations will be apparent to those skilled in the art.Accordingly, it is intended to embrace all such alternatives,modifications and variations that fall within the spirit and broad scopeof the appended claims. All publications, patents and patentapplications mentioned in this specification are herein incorporated intheir entirety by reference into the specification, to the same extentas if each individual publication, patent or patent application wasspecifically and individually indicated to be incorporated herein byreference. In addition, citation or identification of any reference inthis application shall not be construed as an admission that suchreference is available as prior art to the present invention.

1. Apparatus for matching between geometric bodies based on3-dimensional data comprising: an input for receiving representations ofgeometric bodies as Euclidean representations of sets of geodesicdistances between sampled points of a triangulated manifold, saidEuclidean representations being substantially bending invariantrepresentations, a distance calculator for calculating distances betweenrespective geometric bodies based on said Euclidean representation,wherein said distance calculator comprises: an eigenvalue extractor forextracting a predetermined number of eigenvalues from said Euclideanrepresentations, and a plotter for plotting said predetermined number ofeigenvalues as a point on a feature space having a dimension for each ofsaid predetermined number of Eigenvalues, and a thresholder forthresholding a calculated distance to determine the presence or absenceof a match wherein said thresholder is configured to be sensitive toclustering within said feature space.
 2. Apparatus according to claim 1,wherein said predetermined number is three.
 3. Apparatus according toclaim 1, wherein said Euclidean representation is based upon geodesicdistances between a sub-sampling of points of said triangulatedmanifold.
 4. Apparatus according to claim 1, wherein said geometric bodyis a face, having soft geometric regions, being regions susceptible toshort term geometric change and hard geometric regions, being regionssubstantially unsusceptible to short term geometric changes, and whereinsaid Euclidean representation is substantially limited to said hardgeometric regions.
 5. Apparatus according to claim 1, wherein saiddistance calculator is configured to use the Hausdorff metric. 6.Apparatus for obtaining 3-Dimensional data of geometric body formatching, and using said data to carry out matching between differentbodies, said apparatus comprising: a three dimensional scanner forobtaining three-dimensional topographical data of said body, atriangulator for receiving said three-dimensional topographical data ofsaid geometrc body and forming said data into a triangulated manifold, ageodesic converter, connected subsequently to said triangulator, forconverting said triangulated manifold into a series of geodesicdistances between pairs of points of said manifold, a bending invariantmulti-dimensional scalar, connected subsequently to said geodesicconverter, for forming a bending invariant low dimensional Euclideanrepresentation of said series of geodesic distances, said lowdimensional Euclidean representation providing a bending invariantrepresentation of said geometric body, a distance calculator, connectedsubsequently to said bending invariant multi-dimensional scalar, forcalculating distances between geometric bodies based on said Euclideanrepresentation, said distance calculator comprises: an eigenvalueextractor for extracting a predetermined number of eigenvalues from saidEuclidean representations, and a plotter for plotting said predeterminednumber of Eigenvalues as a point on a feature space having a dimensionfor each of said predetermined number of eigenvalues, and a thresholder,connected subsequently to said distance calculator, for thresholding acalculated distance to determine the presence or absence of a match,said thresholder is configured to be sensitive to clustering within saidfeature space, thereby to determine said presence or absence of saidmatch.
 7. Apparatus according to claim 6, wherein said predeterminednumber is three.
 8. Apparatus according to claim 6, wherein saidpredetermined number is greater than three.